A Practical Approach to Understanding Real-World Study Methodology in Cancer Research

00:00:00: You are listening to an ADIS Journal podcast.

00:00:11: Hello and welcome to the second part of the three-part series on practical approaches to understanding real-world evidence published by Oncology and Therapy.

00:00:20: My name is Adam Brusky and I am a professor of medicine at the University of Pittsburgh, the co-director of the Cancer Therapeutics Program of the UPMC-Hillman Cancer Center in Pittsburgh, Pennsylvania.

00:00:32: Today I am joined by Dr.

00:00:33: Winston Chung as we explore real-world study design principles and some of the statistical approaches commonly used in these studies.

00:00:42: Thanks, Adam.

00:00:43: It's great to be here to discuss these important concepts.

00:00:46: My name is Winston Chung, and I'm a professor at the University of Calgary's Cummings School of Medicine and a practicing oncologist in the R&E Charbonneau Cancer Institute.

00:00:56: In part one, we discussed how real-world evidence can complement knowledge gained from randomized controlled trials or RCTs.

00:01:04: We also talked about the evolution of real-world evidence generation into the modern rigorous studies that are prevalent today.

00:01:11: In this podcast, we will be discussing key components of scientific rigor, including the development and adoption of sound study design principles and the appropriate use statistical techniques.

00:01:24: Winston, perhaps you can get us started with a brief overview of real-world study design principles.

00:01:30: Absolutely.

00:01:31: As of any line of scientific inquiry, The research questions should guide the design of the study, and rural studies are no exception.

00:01:39: Rural studies may be retrospective, prospective, or cross-sectional in nature, each with their own strengths and limitations, and each suited to answer different questions.

00:01:50: Retrospective studies are the most common and I think the most relevant for today's discussion.

00:01:56: These can be used to identify correlations between clinical outcomes and treatments that have been approved for a while, as well as to describe dream patterns.

00:02:04: Some commonly used endpoints in real-world oncology research include real-world progression for survival, overall survival, time to chemotherapy, and time to treatment discontinuation.

00:02:18: We should mention that these endpoints, while conceptually similar to their RCT counterparts, differ in some important ways.

00:02:26: For example, for real-world PFS, imaging schedules may vary among practices, and Reese's criteria may not be used as would be the case in RCT.

00:02:39: Furthermore, researchers make choose surrogate time-to-event endpoints, such as time-to-treatment discontinuation.

00:02:46: But the rationale should be clearly communicated.

00:02:50: Retrospective studies require an existing data source.

00:02:53: Some common examples include electronic health records, or EHRs, claims databases, disease registries, or even survey data from physicians and patients.

00:03:05: There are key differences in the quality of these data sources and publications based on these sources should be transparent about their strengths and limitations.

00:03:15: Thanks, Winston.

00:03:16: So for example, if clinicians wanted to learn more about whether a treatment is effective in a group that is not well represented in an RCT, such as elderly patients or a certain racial minority group, a retrospective study using data from a curated EHR database could be appropriate.

00:03:34: Some key advantages to this design would be the immediate available of leave data and the potential for a large sample size to capture sufficient numbers of the patients of interest.

00:03:45: Designs of that type have become more prevalent in recent years and are useful for helping fill knowledge gaps for various patient groups.

00:03:53: However, they do have some limitations.

00:03:55: For most, patients are not randomly assigned to treatments in rural studies.

00:04:00: and patients of various comorbidities who are often intentionally included in these studies, whereas they would be excluded from RCTs.

00:04:08: As such, identifying causal links between treatments and outcomes is far more challenging than in RCTs.

00:04:16: When database sources are used, other limitations arise from the databases themselves, as these may capture a limited number of variables, and there is the possibility of missing, incomplete, or erroneous data points.

00:04:30: Those are important points to keep in mind, but at the same time, I think we should emphasize that there are a number of high-quality sources that serve as foundations for real-world studies.

00:04:40: An example of a well-respected data source is the Surveillance, Epidemiology, and Results, or CIR program of the National Cancer Institute.

00:04:48: The CIR program compiles data from population-based cancer registries that cover almost half of the U.S.

00:04:54: population and has active protocols in place to maintain and enhance data quality.

00:05:00: Another high-quality source has been used in research that I've been involved with is the Flatiron Database, which is a large curated database that aggregates health records for millions of patients who have cancer.

00:05:13: Great points, Adam.

00:05:15: Overall, high-quality studies will provide details on the data source used, such as quality control steps taken by database curators, whether the data were cross-checked against other well-established sources, and its strengths and limitations.

00:05:31: All of these elements can help leaders be more confident in the conclusions drawn from such studies.

00:05:37: As we discussed in part one and touched on here, the primary limitation of studies using real-world data is the lack of randomization of patients to treatment groups.

00:05:47: Randomization in RCTs is crucial to ensure that treatment and control groups differ as little as possible in baseline characteristics, helping to establish a causal link between treatment and outcome.

00:06:01: Unlike RCTs, in routine clinical practice, patient characteristics may influence what treatment they are prescribed.

00:06:08: This is known as treatment selection bias.

00:06:11: And as a result, differences in outcomes may be due to variables other than the actual treatment being studied.

00:06:17: These variables, such as age, sex, or disease status, are known as confounders.

00:06:23: You're right, Adam.

00:06:25: Fortunately, advanced statistical methods are used to help mitigate the bias due to the lack of randomization in real-world studies.

00:06:32: These well-established techniques help balance page demographics and disease characteristics between the study arms and help establish associations between treatments and outcomes.

00:06:45: Some of the statistical tools that are commonly employed by real-world studies include propensity score matching or PSM, inverse probability of treatment weighting or IPTW, and multivariate analysis.

00:06:58: Winston, would you like to give us a brief overview of these methods?

00:07:03: I'd be happy to.

00:07:05: Prophecy score matching is perhaps the most intuitive of these tools.

00:07:09: In this method, each patient from one treatment arm is matched to a patient from the other arm.

00:07:15: To do this, a prophecy score is calculated for each patient, which basically indicates how likely that patient is to receive treatment, given their demographic and clinical characteristics, such as their age, race, ethnicity, location, disease type, etc.

00:07:35: Then, patients are matched based on the Propezi score, where the patients in the experimental group are matched to the most similar patients in the control group.

00:07:44: Patients that cannot be matched are excluded from the analysis.

00:07:48: Following Propezi score matching, typical time-to-event analyses, such as Kauffman-Meier curves or Cox proportional hazardous models, can be conducted to evaluate the effects of treatments on clinical outcomes such as overall survival, or progression-free survival.

00:08:04: It is important to note that because some patients will not be matched to the unique demographic and disease characteristics, the sample size after a propensity score matching will usually be smaller than the unadjusted population.

00:08:17: Most real-world studies are designed in such a way as to ensure enough patients are included in the analysis to have meaningful results.

00:08:25: So, Winston, how does inverse probability of treatment weighting differ from propensity score matching?

00:08:31: IPTW is another technique for balancing study arms.

00:08:36: Like propensity score matching, IPTW starts with computing a propensity score for each patient.

00:08:43: However, in this method, a weight is applied to each patient.

00:08:47: Effectively, if a patient with a particular characteristic rarely receives a treatment, they are up-weighted by a greater amount than those who commonly receive that treatment.

00:08:57: These weights are used to create a so-called pseudo-population, where the distribution of baseline demographics and disease characteristics are similar across study groups.

00:09:10: To make sure the two main groups are well-balanced following potency score matching or IPTW, each baseline or disease characteristic difference needs to be compared between the two true-in groups.

00:09:22: This is called the standardized difference.

00:09:25: If the standardized difference is less than zero point two for each characteristic, then the groups are considered well-balanced.

00:09:32: Rigorous studies typically include these numbers in the baseline demographics table.

00:09:37: That's a great overview of these methods.

00:09:40: As for the relative strengths and weaknesses, propensity score matching might be preferable in a situation where there's a need to study directly matched individuals.

00:09:48: While IPTW could be better suited for a scenario where using the entire population for analysis is desired.

00:09:56: Both techniques are methodologically and statistically sound and I have used them extensively in my own research.

00:10:02: In fact, some studies use both techniques, with one designed as the primary analysis and the other as a sensitivity analysis.

00:10:11: We should also mention that our some instances where neither propensity score matching nor IPTW are the most appropriate test, for example when sample sizes are too small.

00:10:23: In these cases, multivariate analysis may be used.

00:10:26: An example might be the use of multiple regression.

00:10:29: to assess the relative impact of various disease characteristics on a health outcome.

00:10:35: So putting this all together, how can a reader recognize rigor in a real-world study?

00:10:41: I think four key aspects are necessary for well-conducted study.

00:10:46: First, the design should be appropriate for the research question being asked.

00:10:50: Second, the study should use a high-quality database.

00:10:54: Third, there should be a clear description of the statistical methods being employed to reduce confounding.

00:10:59: And finally, the strengths and limitations of the study should be clearly stated.

00:11:03: I

00:11:05: agree that those elements comprise a high-quality real-world study.

00:11:09: The use of trustworthy data sources and well-accepted statistical techniques to balance arms have been key steps in the evolution of real-world evidence generation.

00:11:19: However, I think it is essential to reiterate that even with advanced statistical tools and careful interpretation of the results, real-world studies are not a replacement.

00:11:30: or RCTs.

00:11:32: That's an excellent point and an appropriate place to wrap up today's discussion.

00:11:36: We recognize that statistics used in modern real-world studies can be somewhat intimidating, but we also hope that today's discussion has made them a bit more approachable.

00:11:47: My thanks to Winston for joining me today and helping to bring clarity to this challenging topic.

00:11:53: We hope you stay tuned for the third and final part of the series in which my co-host and I will share our personal journeys who are becoming real-world data advocates, as well as some practical examples of how real-world evidence can support decision-making in the clinic.

00:12:16: You can listen to more podcasts by subscribing to ADIS Journal Podcasts with your preferred podcast provider or by visiting the journal website.

00:12:26: For a full list of declarations, including funding and author disclosure statements and copyright information, please visit the article page on the journal website.

00:12:36: The link to the article page can be found in the podcast description.